Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to repos to maintaned pkgs.k8s.io #657

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Conversation

deric
Copy link
Collaborator

@deric deric commented Sep 22, 2023

Summary

New Kubernetes releases will be published to community owned repost pkgs.k8s.io instead of Google hosted repos.

  • up to v1.23 available only in packages.cloud.google.com
  • v1.24 to v1.27 can be installed from both repos
  • v1.28 onward should be installed from pkgs.k8s.io

Added tests to ensure backwards compatibility.

Related Issues (if any)

Checklist

  • 🟢 Spec tests.
  • 🟢 Acceptance tests.
  • Manually verified. (For example puppet apply)

@deric deric requested review from bastelfreak and a team as code owners September 22, 2023 08:18
@deric deric force-pushed the pkgs branch 2 times, most recently from db17097 to 53b786c Compare September 22, 2023 08:35
manifests/repos.pp Outdated Show resolved Hide resolved
manifests/repos.pp Outdated Show resolved Hide resolved
@deric deric force-pushed the pkgs branch 2 times, most recently from fcd52f8 to eb3f67c Compare September 25, 2023 08:32
@deric
Copy link
Collaborator Author

deric commented Sep 25, 2023

The apt source must have following definition:

deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.25/deb /

Because otherwise we'd get following errors (as CDN is being used):

Notice: /Stage[main]/Apt::Update/Exec[apt_update]/returns: W: GPG error: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.25/deb  InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 234654DA9A296436
Notice: /Stage[main]/Apt::Update/Exec[apt_update]/returns: E: The repository 'https://pkgs.k8s.io/core:/stable:/v1.25/deb  InRelease' is not signed.

The absence of $repos for Debian is weird, because packages are reported to come from an unknown source.

The param key

apt::source { 'kubernetes':
   ...
   key => {}
}

and keyring are mutually exclusive.

apt::source { 'kubernetes':
   ...
   keyring => '',
}

I'm not sure if we can write this in a better, backward compatible way 🤔

@XSpielinbox
Copy link

The apt source must have following definition:

deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.25/deb /

Because otherwise we'd get following errors (as CDN is being used):

Notice: /Stage[main]/Apt::Update/Exec[apt_update]/returns: W: GPG error: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.25/deb  InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 234654DA9A296436
Notice: /Stage[main]/Apt::Update/Exec[apt_update]/returns: E: The repository 'https://pkgs.k8s.io/core:/stable:/v1.25/deb  InRelease' is not signed.

The absence of $repos for Debian is weird, because packages are reported to come from an unknown source.

The param key

apt::source { 'kubernetes':
   ...
   key => {}
}

and keyring are mutually exclusive.

apt::source { 'kubernetes':
   ...
   keyring => '',
}

I'm not sure if we can write this in a better, backward compatible way 🤔

The signed-by option should be backwards compatible with the very old Debian 9 and Ubuntu 16.04 - that should be more than enough in my opinion. The newest versions of Debian and Ubuntu don't work with the old apt-key. Therefore I would definitely opt for the newer variant. See also: puppetlabs/puppetlabs-apt#1034

manifests/repos.pp Outdated Show resolved Hide resolved
@danifr
Copy link
Contributor

danifr commented Oct 19, 2023

Thanks for working on this, I am also interested in having it merged as soon as possible.

For RedHat based OS, would it be possible to also add exclude in the repo definition?

Like in the official example provided here: https://kubernetes.io/blog/2023/08/15/pkgs-k8s-io-introduction/#how-to-migrate-rpm

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

Thank you

@deric
Copy link
Collaborator Author

deric commented Oct 19, 2023

@danifr Sure. Just curious, what does the exclude mean? See baea6f9

@deric deric requested a review from XSpielinbox October 19, 2023 18:15
@XSpielinbox
Copy link

@danifr Sure. Just curious, what does the exclude mean? See baea6f9

As far as I can find and understand it, this excludes updates or installs of these packages from this repository. It does not really make sense to me, however, as this would be the main packages you want to install from this repo or what am I missing?
I haven't tried the effect of this option, but the official documentation and man page for Fedora Linux 38 / RHEL 9 don't reference this option at all.

Sources I found about this option: 1, 2, 3, 4

@danifr
Copy link
Contributor

danifr commented Oct 20, 2023

Yeah disregard my previous comment. My idea was to exclude them from the repo definition so they won't be accidentally upgraded during a regular OS update.

But it seems that we exclude them there, puppet cannot installed them either.

Not sure why it is part of the configuration example...

@deric
Copy link
Collaborator Author

deric commented Oct 20, 2023

@danifr You might be able to use something like yum versionlock kubelet. But I think the whole point of having minor version in repo url:

https://pkgs.k8s.io/core:/stable:/v1.28/rpm/

is that the regular update could upgrade only to next patch version. Which also not ideal because the upgrade should be applied to control-plane nodes first. That's probably the reason why k8s packages are excluded, and when performing manual upgrade you should do this:

yum list --showduplicates kubeadm --disableexcludes=kubernetes

@treydock
Copy link
Contributor

I think the recommendation by Kubernetes to exclude the Kubernetes packages is counterintuitive when configuration management is involved. To use the excludes the only real option with Puppet is install_options on the Package resources to force passing the --disableexcludes=kubernetes flag on install. Generally if someone is doing yum updates they need to do yum update --disablerepo=kubernetes unless they do intend to update Kubernetes since Kubernetes upgrades, even patch releases, require many extra steps.

sorrison added a commit to NeCTAR-RC/puppetlabs-kubernetes that referenced this pull request Jan 15, 2024
Node this is a cherry pick of certain parts of
puppetlabs#657

This is stalled due to yum issues and so doing this quick fix for
now to get us going. Once above is merged we can switch back
to upstream.
@deric deric requested review from GSPatton and removed request for XSpielinbox February 8, 2024 10:17
@jadestorm
Copy link

It looks like this has developed into a more serious issue now -- the google cloud paths are no longer providing content. I imagine they got cleaned up.

@XSpielinbox
Copy link

It looks like this has developed into a more serious issue now -- the google cloud paths are no longer providing content. I imagine they got cleaned up.

As announced back in August, the legacy repos have been frozen for month now and all Kubernetes version that where available through the old repos are long end of life now. As the end of February has passed the old repos are no longer available and if you are still running any version that was available in the old repos you should be looking into upgrading to a supported version as soon as possible.

[1] https://kubernetes.io/blog/2023/08/15/pkgs-k8s-io-introduction/
[2] https://kubernetes.io/blog/2023/08/31/legacy-package-repository-deprecation/
[3] https://kubernetes.io/releases/patch-releases/

@deric deric requested a review from XSpielinbox March 8, 2024 11:45
Copy link

@XSpielinbox XSpielinbox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you decide that you don't want to address my suggested changes soon I can also approve the PR as is.
Just please move forward as soon as possible. Kubernetes is moving fast and has very frequent releases. For the 1.28 branch 1.28.7 already released.

Comment on lines +19 to +20
kubernetes_version => '1.28.2',
kubernetes_package_version => '1.28.2-1.1',

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As 1.29.2 is already out by now this probably should get updated to that new version.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, this is just version for tests. The point was to update the version, that is available in the new repos. We should definitely update the default version, because the current value is 1.10.2.

Comment on lines +88 to +103
# For pkgs.k8s.io use GPG siging key
$_keyring = '/usr/share/keyrings/kubernetes-apt-keyring.gpg'
# TODO: Switch to apt::keyring once supported by puppetlabs-apt
# see: https://github.com/puppetlabs/puppetlabs-apt/pull/1128
archive { '/tmp/kubernetes-apt-keyring.gpg':
source => "https://pkgs.k8s.io/core:/stable:/v${minor_version}/deb/Release.key",
extract => true,
extract_path => '/usr/share/keyrings',
extract_command => 'gpg --dearmor < %s > kubernetes-apt-keyring.gpg',
creates => $_keyring,
}

Apt::Source<| title == 'kubernetes' |> {
keyring => $_keyring,
require => Archive['/tmp/kubernetes-apt-keyring.gpg'],
}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The support for modern keyrings has been released in puppetlabs-apt since three month now in version 9.2. Version 9.4 is also already out. To me, it would make sense to switch to apt::keyring, but if you would like to keep it like this for compatibility for now this would also be fine for me.

The most important for me would be that this PR finally get's merged and there is a release of puppetlabs-kubernetes that has useful, modern defaults and works with the newest version of Kubernetes out of the box.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I wrote earlier, it would be very nice to switch to modern apt::keyring. But we should make incremental changes. Adding dependency on puppetlabs-apt >= 9.2.0 means effectively bumping dependency on puppetlabs-stdlib from >= 4.25.0 to >= 9.0.0. This is rather huge step for the mankind, especially done in a single innocent PR called "Swich repos...".

Depending on puppet code base setup this might mean upgrading many (10 or 40) modules just because of this minor change. I would prefer to release this "old but working version" as a 8.1.0 version. And then change module dependencies in separate PR, while bumping the module version e.g. to 9.0.0.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned in #681, the actual puppetlabs/stdlib requirement is currently >= 8.4.0. So the change wouldn't be that huge.

Comment on lines 14 to 17
full: '16.04'
},
distro: {
codename: 'xenial'

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should perhaps also get updated to a newer version like 22.04, as 24.04 will soon be released, 16.04 is long EOL now and it doesn't help me at all, if it works on some ancient release, but not a current one.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has nothing to do with newer distributions. Google was releasing all packages under xenial codename because the binary packages weren't dependent any specific distribution.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I know that everything was released under the kubernetes-xenial distribution (release in puppet), but I am not completely mistaken this is what got removed as default in https://github.com/puppetlabs/puppetlabs-kubernetes/pull/657/files#diff-b27acae942df5d6ec2b58798add49fff0f625bec33093d74ea3cb1ac39fd0c6aL69, https://github.com/puppetlabs/puppetlabs-kubernetes/pull/657/files#diff-6ff1ce06229c83037f537d2d0cb33aeba339e8fe801238cfb51c2f538c01c114L26 and tested against in https://github.com/puppetlabs/puppetlabs-kubernetes/pull/657/files#diff-6ff1ce06229c83037f537d2d0cb33aeba339e8fe801238cfb51c2f538c01c114L50
but the standalone xenial should refer to the distro, not the kubernetes-repo.

Comment on lines 31 to 37
'kubernetes_yum_baseurl' => 'https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64',
'kubernetes_yum_gpgkey' => 'https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg',
'docker_apt_location' => 'https://download.docker.com/linux/ubuntu',
'docker_apt_release' => 'xenial',
'docker_apt_repos' => 'main',
'docker_yum_baseurl' => 'https://download.docker.com/linux/centos/7/x86_64/stable',
'docker_yum_gpgkey' => 'https://download.docker.com/linux/centos/gpg',

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here: CentOS 7 / RHEL 7 will soon be EOL - CentOS Stream 9 / RHEL 9 have been out for almost 2 years now. For sure it would be good, if it still works on older releases, but people should be moving away from them anyway and ensuring that it works on newer versions is crucial to not hold people back and force them to stick with increasingly insecure OS versions.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I've copy pasted those tests. But these test are garbage in & garbage out tests. It doesn't actually test these distributions. It only tests string getting passed to the class, that it gets propagated. Actual tests should be written in acceptance tests.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah sure, you take some (more or less arbitrary base os) and then test, if 1. If you don't specify a specific repository, it now correctly defaults to the new repo and 2. If you override it with something (e.g. the old repo), it still includes the specified repo as desired, to e.g. still use the old repo for whatever reason.

As this could in theory be OS version dependent, this should in my opinion be tested on one recent release (Ubuntu 22.04 (as you changed it now - thank you!) or Debian 12 and CentOS Stream 9 / RHEL 9) and optionally additionally on some older version as well.

https://github.com/puppetlabs/puppetlabs-kubernetes/pull/657/files#diff-6ff1ce06229c83037f537d2d0cb33aeba339e8fe801238cfb51c2f538c01c114R5-R70 does 1. as far as I understand it: when testing on (now) Ubuntu 22.04 with Kubernetes version 1.28, no additional kubernetes_apt_* and for whatever reason questionable kubernetes_yum_* config, it correctly set's the apt source to the new 1.28 repo.
https://github.com/puppetlabs/puppetlabs-kubernetes/pull/657/files#diff-6ff1ce06229c83037f537d2d0cb33aeba339e8fe801238cfb51c2f538c01c114R71-R132 does 2., testing that on a Debian-based distro when overriding the repo, it set's it to the override.

https://github.com/puppetlabs/puppetlabs-kubernetes/pull/657/files#diff-6ff1ce06229c83037f537d2d0cb33aeba339e8fe801238cfb51c2f538c01c114R195-R246 tests 1., that on CentOS 7 the new default works.
https://github.com/puppetlabs/puppetlabs-kubernetes/pull/657/files#diff-6ff1ce06229c83037f537d2d0cb33aeba339e8fe801238cfb51c2f538c01c114L127-R297 tests 2., that the override works for CentOS 7, while still configuring some questionable apt stuff, that should be absolutely irrelevant on dnf-based distros.

And finally https://github.com/puppetlabs/puppetlabs-kubernetes/pull/657/files#diff-6ff1ce06229c83037f537d2d0cb33aeba339e8fe801238cfb51c2f538c01c114L127-R297 does 2. again on CentOS 7, but now with packaged containerd instead of docker archives.

Comment on lines +200 to +226
operatingsystemrelease: '7.0',
kernel: 'Linux',
os: {
family: 'RedHat',
name: 'RedHat',
release: {
full: '7.0'
}
}
}
end

let(:params) do
{
'container_runtime' => 'docker',
'kubernetes_version' => '1.28.1',
'kubernetes_apt_location' => '',
'kubernetes_apt_release' => '',
'kubernetes_apt_repos' => '',
'kubernetes_key_id' => '',
'kubernetes_key_source' => '',
'kubernetes_yum_baseurl' => '',
'kubernetes_yum_gpgkey' => '',
'docker_apt_location' => 'https://download.docker.com/linux/ubuntu',
'docker_apt_release' => 'xenial',
'docker_apt_repos' => 'main',
'docker_yum_baseurl' => 'https://download.docker.com/linux/centos/7/x86_64/stable',

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above

@jadestorm
Copy link

Just wanted to add a couple of notes --

  1. It is possible to pass in (via hiera or direct) some reasonable settings to get the latest release, as-is, to use pkgs.k8s.io -- we are actively using that right now because ...
  2. I can't explain why but when I used your fork for this PR as-is @deric , I got the following dependency loop which frankly makes zero sense to me after looking through the code:
    Error: Found 1 dependency cycle:
    (Exec[Install cni network provider] => Exec[Install cni network provider])\nTry the '--graph' option and opening the resulting '.dot' file in OmniGraffle or GraphViz
    Error: Failed to apply catalog: One or more resource dependency cycles detected in graph

I could not track down specifically what I might have been doing to cause this, if anything, but I just wanted to mention it. It also did not occur on ALL of our hosts using this module -- so there's a good chance it's something we caused. None-the-less, it does not occur in 8.0.0 so I wanted to bring it up not as a problem with this PR, but as a note. I believe your fork only has work-since-8.0.0 plus pkgs changes.

I can share the hiera method of working around this issue with 8.0.0 if anyone needs it.

@XSpielinbox
Copy link

1. It is possible to pass in (via hiera or direct) some reasonable settings to get the latest release, as-is, to use pkgs.k8s.io -- we are actively using that right now because ...

@jadestorm from looking at the code something like setting

kubernetes::kubernetes_apt_location: "https://pkgs.k8s.io/core:/stable:/v${minor_version}/deb"
kubernetes::kubernetes_apt_release: '/'
kubernetes::kubernetes_version: 1.29.2
kubernetes::kubernetes_package_version: 1.29.2-1.1
kubernetes::kubernetes_apt_repos: ''
kubernetes::kubernetes_key_id: '<the new fingerprint>'
kubernetes::kubernetes_key_source: "https://pkgs.k8s.io/core:/stable:/v${minor_version}/deb/Release.key"

in the hiera should work (given that your containerd, systemd, etc. config is updated already) , but haven't tested it directly.
${minor_version} of course must be substituted either directly or by doing something like

$kubernetes_version = lookup('kubernetes::kubernetes_version')

$parts = split($kubernetes_version, '[.]')
$minor_version = "${parts[0]}.${parts[1]}"

waipeng pushed a commit to NeCTAR-RC/puppetlabs-kubernetes that referenced this pull request May 24, 2024
Node this is a cherry pick of certain parts of
puppetlabs#657

This is stalled due to yum issues and so doing this quick fix for
now to get us going. Once above is merged we can switch back
to upstream.

(cherry picked from commit 07acbc1)
nectar-gerrit pushed a commit to NeCTAR-RC/puppetlabs-kubernetes that referenced this pull request May 28, 2024
Node this is a cherry pick of certain parts of
puppetlabs#657

This is stalled due to yum issues and so doing this quick fix for
now to get us going. Once above is merged we can switch back
to upstream.

(cherry picked from commit 07acbc1)
Change-Id: I452a7e087cda5bb0089b5faa7b14c6c633ed1497
@bilbof
Copy link

bilbof commented Oct 4, 2024

FYI for folks blocked, my workaround on Ubuntu 24.04 was:

kubernetes::kubernetes_version: 1.31.1
kubernetes::kubernetes_package_version: 1.31.1-1.1
kubernetes::create_repos: false

# you dont need these
# kubernetes::kubernetes_apt_release: "kubernetes-%{os.distro.codename}" # kubernetes-noble
# kubernetes::kubernetes_apt_location: 'https://pkgs.k8s.io'
# kubernetes::kubernetes_apt_repos: ""
# gpg --show-keys <(curl https://packages.cloud.google.com/apt/doc/apt-key.gpg)
# kubernetes::kubernetes_key_id: '35BAA0B33E9EB396F59CA838C0BA5CE6DC6315A3'
# kubernetes::kubernetes_key_source: 'https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key'
# kubernetes::docker_package_name: 'docker-ce'
# run before puppetlabs-kubernetes

apt::source { 'kubernetes':
    location => 'https://pkgs.k8s.io/core:/stable:/v1.31/deb/',
    repos    => '',
    release  => '/',
    key      => {
      name => 'kubernetes-apt-keyring.gpg',
      source => 'https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key'
    },
    include     => {
      'signed-by' => '/etc/apt/keyrings/kubernetes-apt-keyring.gpg',
    }
  }

  apt::source { 'docker':
    location => 'https://download.docker.com/linux/ubuntu',
    repos    => '',
    release  => 'noble stable',
    key      => {
      'name'     => 'docker.asc',
      'source' => 'https://download.docker.com/linux/ubuntu/gpg',
    },
    include     => {
      'arch'      => 'amd64',
      'signed-by' => '/etc/apt/keyrings/docker.asc',
    }
  }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants